Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-26959 Brotli compression support #4353

Merged
merged 9 commits into from
Apr 22, 2022
Merged

Conversation

apurtell
Copy link
Contributor

@apurtell apurtell commented Apr 17, 2022

I was about to document the compression improvements upcoming in 2.5.0 and realized we lacked support for one of the major compression formats in use today. (Arguably brotli is not often used to compress data at rest, but is widely supported among web browsers and HTTP servers and clients.)

Brotli support for Java is provided by Apache 2 licensed Brotli4j available in Maven central.

It is not difficult to add it, so let's add it. There might be some advantage to opting for brotli compression when storing web data given the precomputed static dictionary of such content included in its specification.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 41s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 43s Maven dependency ordering for branch
+1 💚 mvninstall 2m 20s master passed
+1 💚 compile 5m 53s master passed
+1 💚 checkstyle 1m 7s master passed
+1 💚 spotbugs 7m 48s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for patch
+1 💚 mvninstall 2m 7s the patch passed
+1 💚 compile 5m 54s the patch passed
-0 ⚠️ javac 5m 54s root generated 2 new + 1470 unchanged - 2 fixed = 1472 total (was 1472)
-0 ⚠️ checkstyle 1m 6s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 12m 1s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚 spotbugs 8m 31s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
54m 36s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4353
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml
uname Linux 9dc9603ff112 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/artifact/yetus-general-check/output/diff-compile-javac-root.txt
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt
Max. process+thread count 138 (vs. ulimit of 30000)
modules C: hbase-common hbase-compression hbase-compression/hbase-compression-brotli . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 22s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for branch
+1 💚 mvninstall 4m 24s master passed
+1 💚 compile 3m 12s master passed
+1 💚 shadedjars 5m 59s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 4m 16s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 4m 21s the patch passed
+1 💚 compile 3m 21s the patch passed
+1 💚 javac 3m 21s the patch passed
+1 💚 shadedjars 6m 3s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 4m 31s the patch passed
_ Other Tests _
-1 ❌ unit 271m 11s root in the patch failed.
311m 37s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux f3a22061cbc4 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/testReport/
Max. process+thread count 2634 (vs. ulimit of 30000)
modules C: hbase-common hbase-compression hbase-compression/hbase-compression-brotli . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 3s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 18s Maven dependency ordering for branch
+1 💚 mvninstall 2m 19s master passed
+1 💚 compile 1m 35s master passed
+1 💚 shadedjars 3m 42s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 56s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 2m 36s the patch passed
+1 💚 compile 1m 44s the patch passed
+1 💚 javac 1m 44s the patch passed
+1 💚 shadedjars 3m 56s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 11s the patch passed
_ Other Tests _
+1 💚 unit 347m 31s root in the patch passed.
371m 10s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8f0dc6397cf0 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/testReport/
Max. process+thread count 4562 (vs. ulimit of 30000)
modules C: hbase-common hbase-compression hbase-compression/hbase-compression-brotli . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtell
Copy link
Contributor Author

Rebase and add supplemental models.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 41s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for branch
+1 💚 mvninstall 2m 6s master passed
+1 💚 compile 5m 51s master passed
+1 💚 checkstyle 1m 7s master passed
+1 💚 spotbugs 7m 57s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
-1 ❌ mvninstall 2m 3s root in the patch failed.
-1 ❌ compile 5m 51s root in the patch failed.
-0 ⚠️ javac 5m 51s root in the patch failed.
-0 ⚠️ checkstyle 1m 5s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 3s The patch has no ill-formed XML file.
-1 ❌ hadoopcheck 2m 54s The patch causes 10 errors with Hadoop v3.1.2.
-1 ❌ hadoopcheck 5m 55s The patch causes 10 errors with Hadoop v3.2.2.
-1 ❌ hadoopcheck 8m 55s The patch causes 10 errors with Hadoop v3.3.1.
-1 ❌ spotbugs 7m 1s root in the patch failed.
_ Other Tests _
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
46m 42s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4353
Optional Tests dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle
uname Linux e37fcb4130c3 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-compile-root.txt
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/diff-checkstyle-root.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-javac-3.1.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-javac-3.2.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-javac-3.3.1.txt
spotbugs https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-general-check/output/patch-spotbugs-root.txt
Max. process+thread count 137 (vs. ulimit of 30000)
modules C: hbase-common hbase-compression hbase-compression/hbase-compression-brotli hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 19s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for branch
+1 💚 mvninstall 2m 35s master passed
+1 💚 compile 1m 52s master passed
+1 💚 shadedjars 3m 41s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 38s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
-1 ❌ mvninstall 2m 39s root in the patch failed.
-1 ❌ compile 1m 51s root in the patch failed.
-0 ⚠️ javac 1m 51s root in the patch failed.
+1 💚 shadedjars 3m 40s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 53s the patch passed
_ Other Tests _
-1 ❌ unit 261m 22s root in the patch failed.
287m 31s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux cff28ffe8dc8 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-11.0.10+9
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/testReport/
Max. process+thread count 4558 (vs. ulimit of 30000)
modules C: hbase-common hbase-compression hbase-compression/hbase-compression-brotli . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 8s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for branch
+1 💚 mvninstall 2m 18s master passed
+1 💚 compile 1m 33s master passed
+1 💚 shadedjars 3m 41s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 6s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
-1 ❌ mvninstall 2m 12s root in the patch failed.
-1 ❌ compile 1m 38s root in the patch failed.
-0 ⚠️ javac 1m 38s root in the patch failed.
+1 💚 shadedjars 3m 43s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 17s the patch passed
_ Other Tests _
-1 ❌ unit 367m 54s root in the patch failed.
391m 58s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 2c153a1cba24 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-root.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/testReport/
Max. process+thread count 4526 (vs. ulimit of 30000)
modules C: hbase-common hbase-compression hbase-compression/hbase-compression-brotli hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 18s Maven dependency ordering for branch
+1 💚 mvninstall 2m 48s master passed
+1 💚 compile 7m 16s master passed
+1 💚 checkstyle 1m 16s master passed
+1 💚 spotbugs 9m 41s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
-1 ❌ mvninstall 2m 38s root in the patch failed.
-1 ❌ compile 6m 58s root in the patch failed.
-0 ⚠️ javac 6m 58s root in the patch failed.
+1 💚 checkstyle 1m 21s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 5s The patch has no ill-formed XML file.
-1 ❌ hadoopcheck 3m 37s The patch causes 10 errors with Hadoop v3.1.2.
-1 ❌ hadoopcheck 7m 13s The patch causes 10 errors with Hadoop v3.2.2.
-1 ❌ hadoopcheck 10m 35s The patch causes 10 errors with Hadoop v3.3.1.
-1 ❌ spotbugs 7m 12s root in the patch failed.
_ Other Tests _
+1 💚 asflicense 0m 52s The patch does not generate ASF License warnings.
56m 3s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4353
Optional Tests dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle
uname Linux 70b24e2d0d4b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-compile-root.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-javac-3.1.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-javac-3.2.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-javac-3.3.1.txt
spotbugs https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-general-check/output/patch-spotbugs-root.txt
Max. process+thread count 139 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-brotli . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtell
Copy link
Contributor Author

Precommit failures are all of the form

[INFO] --- maven-enforcer-plugin:3.0.0:enforce (check-aggregate-license) @ hbase-assembly ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed with message:
License errors detected, for more detail find ERROR in
                /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4353/yetus-general-check/src/hbase-assembly/target/maven-shared-archive-resources/META-INF/LICENSE

so I don't have the supplimental models changes quite right yet, even though it seems fine locally.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 43s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 17s Maven dependency ordering for branch
+1 💚 mvninstall 3m 21s master passed
+1 💚 compile 2m 9s master passed
+1 💚 shadedjars 4m 24s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 12s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
-1 ❌ mvninstall 3m 8s root in the patch failed.
-1 ❌ compile 2m 18s root in the patch failed.
-0 ⚠️ javac 2m 18s root in the patch failed.
+1 💚 shadedjars 4m 33s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 38s the patch passed
_ Other Tests _
-1 ❌ unit 243m 42s root in the patch failed.
274m 52s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 56ef0eedb711 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-11.0.10+9
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/testReport/
Max. process+thread count 2596 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-brotli hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtell
Copy link
Contributor Author

The precommit failure is unrelated to this change

[ERROR] org.apache.hadoop.hbase.security.TestSecureIPC.testSaslWithCryptoAES[3:rpcClientImpl=org.apache.hadoop.hbase.ipc.NettyRpcClient, rpcServerImpl=org.apache.hadoop.hbase.ipc.NettyRpcServer]  Time elapsed: 0.141 s  <<< ERROR!
java.lang.NullPointerException

@apurtell
Copy link
Contributor Author

I don't understand the license errors.

License errors detected, for more detail find ERROR in
                /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-4353/yetus-jdk11-hadoop3-check/src/hbase-assembly/target/maven-shared-archive-resources/META-INF/LICENSE

This is NOT failing locally, "mvn clean install assembly:single -DskipTests"

@apurtell
Copy link
Contributor Author

Oh perhaps b436ab4 has not been tested yet. Still shows as yellow.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 13s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 3m 27s master passed
+1 💚 compile 2m 25s master passed
+1 💚 shadedjars 5m 27s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 14s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for patch
-1 ❌ mvninstall 3m 13s root in the patch failed.
-1 ❌ compile 2m 24s root in the patch failed.
-0 ⚠️ javac 2m 24s root in the patch failed.
+1 💚 shadedjars 5m 21s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 28s the patch passed
_ Other Tests _
-1 ❌ unit 425m 18s root in the patch failed.
458m 9s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux e4bb769a217f 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-root.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-root.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/testReport/
Max. process+thread count 2120 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-brotli . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for branch
+1 💚 mvninstall 2m 9s master passed
+1 💚 compile 6m 8s master passed
+1 💚 checkstyle 1m 8s master passed
+1 💚 spotbugs 8m 13s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 2m 8s the patch passed
+1 💚 compile 6m 11s the patch passed
-0 ⚠️ javac 6m 11s root generated 2 new + 1470 unchanged - 2 fixed = 1472 total (was 1472)
-0 ⚠️ checkstyle 1m 9s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 3s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 11m 29s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚 spotbugs 8m 52s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 56s The patch does not generate ASF License warnings.
55m 33s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4353
Optional Tests dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle
uname Linux be9871912918 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/artifact/yetus-general-check/output/diff-compile-javac-root.txt
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/artifact/yetus-general-check/output/diff-checkstyle-root.txt
Max. process+thread count 137 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-brotli hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 39s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 2m 34s master passed
+1 💚 compile 1m 42s master passed
+1 💚 shadedjars 3m 50s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 30s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 2m 25s the patch passed
+1 💚 compile 1m 40s the patch passed
+1 💚 javac 1m 40s the patch passed
+1 💚 shadedjars 3m 45s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 34s the patch passed
_ Other Tests _
+1 💚 unit 227m 18s root in the patch passed.
252m 57s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 1da930b4a39f 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/testReport/
Max. process+thread count 5013 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-brotli . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtell apurtell requested review from Apache9 and ndimiduk April 19, 2022 01:56
@apurtell
Copy link
Contributor Author

I subjected the new codec to the same IntegrationTestLoadCommonCrawl stress test scenario used for the earlier compression work and it passed.

Copy link
Member

@ndimiduk ndimiduk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions and small nits -- otherwise, looks good.

</dependency>
<!-- native Java compression codecs -->
<dependency>
<groupId>com.aayushatharva.brotli4j</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we shade this dependency? Is there a scenario where user code and this jar find themselves on the same class path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have not shaded any other compression support libraries. It could be a follow up issue to do them all if it seems like a good idea.

Copy link
Contributor Author

@apurtell apurtell Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I don't believe shading can work. All of the compression support libraries except for aircompressor (with its raison d'être being pure Java implementation) extract platform specific native shared libraries and load them dynamically at runtime. The symbols compiled on the native side of the JNI bridge cannot be relocated by shading so the result will fail to link. I made sure we only included options that have native support for Linux x86_64 and Linux aarch64. Usually they also have support for MacOS x86_64. That is the case for Brotli4j.

@apurtell
Copy link
Contributor Author

Rebased on latest master, incorporating improvements from review feedback.

@apurtell
Copy link
Contributor Author

apurtell commented Apr 20, 2022

I have an approval and addressed all review feedback. Planning to merge this evening unless objection. Waiting for latest precommit to check before commit.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 3m 22s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for branch
+1 💚 mvninstall 2m 24s master passed
+1 💚 compile 6m 10s master passed
+1 💚 checkstyle 1m 10s master passed
+1 💚 spotbugs 9m 33s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 2m 8s the patch passed
+1 💚 compile 6m 19s the patch passed
-0 ⚠️ javac 6m 19s root generated 75 new + 1389 unchanged - 83 fixed = 1464 total (was 1472)
+1 💚 checkstyle 1m 9s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 3s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 11m 39s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚 spotbugs 10m 55s the patch passed
_ Other Tests _
+1 💚 asflicense 1m 40s The patch does not generate ASF License warnings.
63m 36s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4353
Optional Tests dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle
uname Linux 436c538965b3 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/artifact/yetus-general-check/output/diff-compile-javac-root.txt
Max. process+thread count 139 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 18s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for branch
+1 💚 mvninstall 2m 34s master passed
+1 💚 compile 1m 52s master passed
+1 💚 shadedjars 3m 40s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 43s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 17s Maven dependency ordering for patch
+1 💚 mvninstall 2m 33s the patch passed
+1 💚 compile 1m 54s the patch passed
+1 💚 javac 1m 54s the patch passed
+1 💚 shadedjars 3m 38s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 56s the patch passed
_ Other Tests _
-1 ❌ unit 241m 17s root in the patch failed.
270m 37s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 4d7e4821d6cb 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/testReport/
Max. process+thread count 4383 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. The only concern is about the LICENSE aggregating, let's wait for @busbey 's input? Or mind posting the errors here? In the past we usually get through this by modifying the supplemental model.

Thanks @apurtell for the effort on improving the compression support.

conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT));
return size > 0 ? size : 256 * 1024; // Don't change this default
// IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm, why we have an invalid default value?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is Hadoop's default, IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is a Hadoop constant. We reuse these where we can. But the value of '0' gives an error creating the (Hadoop) compression stream. So we don't accept it here.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 41s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 3m 28s master passed
+1 💚 compile 2m 23s master passed
+1 💚 shadedjars 5m 27s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 4m 21s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for patch
+1 💚 mvninstall 3m 18s the patch passed
+1 💚 compile 2m 24s the patch passed
+1 💚 javac 2m 24s the patch passed
+1 💚 shadedjars 5m 22s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 4m 27s the patch passed
_ Other Tests _
-1 ❌ unit 373m 10s root in the patch failed.
410m 37s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 4fb9e0e237cf 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 35aa57e
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/testReport/
Max. process+thread count 2199 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/5/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtell
Copy link
Contributor Author

apurtell commented Apr 21, 2022

@Apache9 I did not save the error text. It is the standard error text you get when LICENSE.vm cannot be processed. It said to add "apache license 2.0" to non_aggregated_fine if appropriate. So that is what I did, because that is true in this case. The reason this happens is due to the Brotli4j POMs, which do not use the exact text to describe the Apache 2.0 license as expected. Nothing we can do about it except add it to non_aggregated_fine as intended, I believe. I tried to substitute for the Brotli4j model in suppliemental_models but this did not work.

Brotli4j is licensed with the Apache 2.0 license.

@apurtell
Copy link
Contributor Author

apurtell commented Apr 21, 2022

Unless @busbey actually responds soon I have an approval and will merge this to move forward. Let me give it another few days, there is no rush, early next week.

He responded upthread.

@apurtell
Copy link
Contributor Author

After ingesting ~5TB of CommonCrawl data, brotli compressing at level 7 offers a few % improvement (-85% vs -82%) over zstandard also compressing at its notion of level 7. This stands to reason as brotli will fall back on its static dictionary pre-trained on web content before adding input to the generic/dynamic dictionary; while zstandard does not have this advantage (although it could, if the user configures a suitable pre-trained dictionary for it (see HBASE-26353)). At scale that difference would be noticeable.

@apurtell
Copy link
Contributor Author

apurtell commented Apr 21, 2022

Rebased on master, using supplemental models to fix the LICENSE.vm issue as required.

(@busbey pointed me at HBASE-26760 where another workaround like I proposed here was backed out.)

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 4m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for branch
+1 💚 mvninstall 2m 2s master passed
+1 💚 compile 5m 47s master passed
+1 💚 checkstyle 1m 5s master passed
+1 💚 spotbugs 9m 13s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 2m 5s the patch passed
+1 💚 compile 5m 55s the patch passed
-0 ⚠️ javac 5m 55s root generated 15 new + 1458 unchanged - 15 fixed = 1473 total (was 1473)
+1 💚 checkstyle 1m 17s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 4s The patch has no ill-formed XML file.
+1 💚 hadoopcheck 11m 36s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚 spotbugs 10m 27s the patch passed
_ Other Tests _
+1 💚 asflicense 1m 15s The patch does not generate ASF License warnings.
62m 49s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4353
Optional Tests dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle
uname Linux 04a7224a32f7 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 41ae90d
Default Java AdoptOpenJDK-1.8.0_282-b08
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/artifact/yetus-general-check/output/diff-compile-javac-root.txt
Max. process+thread count 138 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 6m 45s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for branch
+1 💚 mvninstall 2m 40s master passed
+1 💚 compile 1m 50s master passed
+1 💚 shadedjars 3m 38s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 43s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 2m 33s the patch passed
+1 💚 compile 1m 51s the patch passed
+1 💚 javac 1m 51s the patch passed
+1 💚 shadedjars 3m 41s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 3m 58s the patch passed
_ Other Tests _
+1 💚 unit 243m 6s root in the patch passed.
277m 44s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 62767ce18835 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 41ae90d
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/testReport/
Max. process+thread count 4526 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd hbase-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 6m 5s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 18s Maven dependency ordering for branch
+1 💚 mvninstall 2m 22s master passed
+1 💚 compile 1m 36s master passed
+1 💚 shadedjars 3m 42s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 2m 59s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 2m 28s the patch passed
+1 💚 compile 1m 51s the patch passed
+1 💚 javac 1m 51s the patch passed
+1 💚 shadedjars 3m 54s patch has no errors when building our shaded downstream artifacts.
-0 ⚠️ javadoc 1m 33s root generated 3 new + 82 unchanged - 3 fixed = 85 total (was 85)
_ Other Tests _
+1 💚 unit 345m 54s root in the patch passed.
377m 34s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4353
Optional Tests javac javadoc unit shadedjars compile
uname Linux 3e624ab43e78 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 41ae90d
Default Java AdoptOpenJDK-1.8.0_282-b08
javadoc https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-root.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/testReport/
Max. process+thread count 4692 (vs. ulimit of 30000)
modules C: hbase-common hbase-resource-bundle hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd . hbase-assembly U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4353/6/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@busbey
Copy link
Contributor

busbey commented Apr 22, 2022

I haven't gotten to dig into the change beyond the licensing stuff. but seems good to go to me given the existing review and this update on license handling.

@apurtell apurtell merged commit f5b10e0 into apache:master Apr 22, 2022
@apurtell apurtell deleted the HBASE-26959 branch April 22, 2022 23:42
asfgit pushed a commit that referenced this pull request Apr 23, 2022
asfgit pushed a commit that referenced this pull request Apr 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants